Skip to content

fix(db): batch signal snapshot target lookups - #3465

Merged
JSONbored merged 1 commit into
mainfrom
codex/propose-fix-for-unbounded-sql-in-list
Jul 5, 2026
Merged

fix(db): batch signal snapshot target lookups#3465
JSONbored merged 1 commit into
mainfrom
codex/propose-fix-for-unbounded-sql-in-list

Conversation

@JSONbored

@JSONbored JSONbored commented Jul 5, 2026

Copy link
Copy Markdown
Owner

Motivation

  • Prevent a scheduled repo-doc refresh fan-out from building an unbounded SQL IN parameter list that can exceed D1/back-end bound-parameter limits and fail the sweep, causing no per-repo refresh jobs to be enqueued.
  • Add a regression test showing the helper splits large target-key sets into bounded queries so the bulk lookup preserves behavior without risking a parameter overflow.

Description

  • Add SIGNAL_SNAPSHOT_TARGET_KEY_SQL_BATCH = 90 and change listLatestSignalSnapshotsForTargets to iterate targetKeys in batches and run one parametrized query per batch so bound-parameter counts stay bounded.
  • Preserve the original latest-per-target semantics (ROW_NUMBER() partition by target_key) and merge batch results into a single returned Map<string, SignalSnapshotRecord>.
  • Add a unit regression in test/unit/data-spine.test.ts that seeds 95 keys, wraps the test DB to capture bound-parameter counts, and asserts the query was split into two batches ([91, 6], accounting for signalType plus targets) while returning all results.
  • Files changed: src/db/repositories.ts and test/unit/data-spine.test.ts.

Testing

  • Ran npx vitest run test/unit/data-spine.test.ts -t "listLatestSignalSnapshotsForTargets" and the updated unit test passed.
  • Ran npm run typecheck and git diff --check which both succeeded.
  • Attempted the full gate with npm run test:ci; the full suite did not complete in this environment due to unrelated long-running queue/backfill tests and external actionlint setup DNS issues, so test:ci did not finish green here.
  • npm audit --audit-level=moderate returned a registry audit endpoint error (403) in this environment and could not be validated here.

Codex Task

@JSONbored JSONbored added the codex label Jul 5, 2026
@superagent-security

superagent-security Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@loopover-orb loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 5, 2026
@codecov

codecov Bot commented Jul 5, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.47%. Comparing base (1ad5b35) to head (75cb63c).
⚠️ Report is 22 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files 
@@           Coverage Diff           @@
##             main    #3465   +/-   ##
=======================================
  Coverage   93.47%   93.47%           
=======================================
  Files         292      292           
  Lines       30797    30800    +3     
  Branches    11225    11225           
=======================================
+ Hits        28786    28789    +3     
  Misses       1355     1355           
  Partials      656      656
Files with missing lines Coverage Δ
src/db/repositories.ts 96.46% <100.00%> (+<0.01%) ⬆️
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@loopover-orb

loopover-orb Bot commented Jul 5, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Warning

🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨

⏸️ Gittensory review result - manual review recommended

Review updated: 2026-07-05 08:21:34 UTC

2 files · 1 AI reviewer · 2 blockers · readiness 93/100 · CI green · clean

⏸️ Suggested Action - Manual Review

  • No linked issue detected — If this PR is intended to solve an issue, link it explicitly in the PR body.
  • Maintainer requires a linked issue — Link the relevant issue (for example Closes #123) before opening the PR.

Review summary
The change correctly bounds `listLatestSignalSnapshotsForTargets` by slicing target keys into 90-key batches while preserving the latest-per-target query shape inside each batch, so the D1 parameter count stays capped at 91 including `signalType`. The regression test exercises the production helper with 95 targets and verifies both the split and returned rows. I do not see a reachable correctness regression in the visible diff.

Nits — 5 non-blocking
  • test/unit/data-spine.test.ts:520 uses generated timestamps like `00:94:00.000Z`, which are not valid ISO times; nit: keep fixtures valid even when ordering does not matter so this test does not normalize a bad timestamp pattern.
  • src/db/repositories.ts:4129 adds a second SQL batch-size constant instead of reusing or colocating with the existing `SQL_IN_BATCH = 90` pattern in RAG code; nit: consider a shared DB helper/constant if this pattern is expected to recur.
  • test/unit/data-spine.test.ts:520 can generate valid timestamps with a fixed date plus seconds or milliseconds derived from `index`, avoiding minute values above 59.
  • src/db/repositories.ts:4129 could add a short comment that the batch size leaves room for the leading `signalType` bind, matching the test expectation of `[91, 6]`.
  • PR description should explicitly close or link the eligible issue for this work; the visible text references `feat(selfhost): scheduled + on-demand refresh for repo-doc generation #3202` as a review finding, but does not clearly state the issue relationship.

Concerns raised — review before merging

  • No linked issue detected — If this PR is intended to solve an issue, link it explicitly in the PR body.
  • Maintainer requires a linked issue — Link the relevant issue (for example Closes #123) before opening the PR.
Signal Result Evidence
Code review ❌ 2 blockers 1 reviewer
Linked issue ⚠️ Missing No linked issue or no-issue rationale found.
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (no linked issue context).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 56 registered-repo PR(s), 46 merged, 586 issue(s).
Contributor context ✅ Confirmed Gittensor contributor JSONbored; Gittensor profile; 56 PR(s), 586 issue(s).
Gate result ❌ Blocking Repo-configured hard blocker found.
Review context
  • Author: JSONbored
  • Role context: owner (maintainer lane)
  • Public audience mode: oss maintainer
  • Lane context: Repository registration is not available in the local Gittensory cache.
  • Public profile languages: not available
  • Official Gittensor activity: 56 PR(s), 586 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Treat this as maintainer-lane context rather than normal contributor-lane activity.
  • Explain no-issue PR.
  • No action.
  • Link the issue being solved, or explicitly explain why this is a no-issue PR.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

  • Re-run Gittensory review

@loopover-orb loopover-orb Bot added the manual-review Gittensor contributor context label Jul 5, 2026
@JSONbored
JSONbored merged commit 3a84119 into main Jul 5, 2026
10 checks passed
@JSONbored
JSONbored deleted the codex/propose-fix-for-unbounded-sql-in-list branch July 5, 2026 08:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@JSONbored JSONbored

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. manual-review Gittensor contributor context

Projects

Self-Hosted Review Stack (LoopOver Orb)
Status: Done

Milestone

Maintainer auto-maintain & convergenc...

Development

Successfully merging this pull request may close these issues.

1 participant

@JSONbored